Connecting Policy and Data: 

What Are Your State's Critical Policy Questions? 



student Achievement 


■ Data Alone Are Not Valuable: Connect Data and Policy 


Central to the 2005 launch of the Data Quality Campaign 
(DQC) was highlighting a set of 28 policy questions 
(Appendix A) that could not be answered without a 
statewide longitudinal data system consisting of 
10 Essential Elements. High-quality longitudinal data 
are valuable only if that information is used to inform 
the pressing policy discussions and practical decisions 
policymakers, educators and other stakeholders face. 

To date, information technology (IT) staff — including 
chief information officers, analysts, and state and 

Working Together 

The following anecdote (told to DQC staff at a state meeting) 
illustrates the inefficiencies and costs that states bear when 
policymakers do not work in partnership with data managers 
to develop data systems. 

A state legislator contacted the data manager at the state 
department of education and asked how many of the state's 
students were currently enrolled in a physical education (PE) 
course. The data manager responded by saying that the state 
does not collect those data. The data manager pushed a bit 
deeper by asking the legislator what exactly he was trying 
to answer. The legisiator responded that his committee 
was looking at student health/wellness/htness issues and 
thought that PE enrollment would be a good indicator. The 
data manager informed him that although the state did not 
collect PE data, it did collect a number of other data elements 
that when taken together and analyzed would likely produce 
the type of information the committee needed to make some 
data-informed decisions. 

This story illustrates how stakeholders have become used to 
relying on proxy indicators because they are not used to 


local data managers — have provided vital leadership 
in developing state longitudinal data systems, but 
policymakers now must take on a more prominent role. 
Building and maintaining data systems is not solely an 
IT project. Policymakers from across the P-20 spectrum 
(e.g., governors, legislators, chiefs, agency heads and 
postsecondary leaders) need to prioritize and elucidate 
the critical questions and issues that data systems must 
answer. All stakeholders, policymakers in particular, need 
to advise the IT team on what data should be collected, 
how the data should be defined, how often they need to be 

having timely and useful access to rich information and that 
data managers and the users of the data often do not work in 
partnership, but in a responsive manner. 

■ Many state and federal data collections are the result 
of a policymaker looking for a piece of information to 
answer a question. In this case, upon learning that the 
state does not collect PE data, the legislator could have 
introduced legislation mandating the collection, which 
would have resulted in a time-consuming (often taking 
several years and increasing the reporting burden on 
districts and states) and expensive effort. By engaging 
the policymaker on the policy problem at hand, the 
data manager was able to leverage existing resources to 
provide the information needed at a much lower cost to 
the state and on a shorter timeline. 

■ This particular data manager either had the necessary 
expertise or had access to the expertise to deliver 
quality, timely and useful information to the legislator. 

The development of analytical capacity around data is 
critical if states are going to successfully connect data 
with policy. 
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collected and how they need to be analyzed. Just as we do 
not ask construction workers to turn our houses into livable 
homes without guidance and input from the homeowner, 
we cannot ask IT staff solely to be responsible for turning 
large databases into robust information systems without 
guidance from policymakers. 

To ensure that the data system infrastructure investments 
are leveraged fully, states must identify what the data are 
going to be used for, not the other way around. Good 
management for any project, including building a data 
system, starts with asking what the end goal of the effort 
is. The end goal in this case is not to have a longitudinal 
P-20 /workforce data system but to have the ability to 
inform policy and practical questions that ensure alignment 
and continuous improvement across the human capital 
development system. 


Policymakers and IT / 
data managers are 
essential partners in 
building, continuing 
to maintain and 
ensuring access to 
these systems. No 
one partner can do 
this alone, and a 
governance policy 
needs to be in place 
to ensure that data 
directors are part of 
policy conversations 

and that policymakers and other stakeholders articulate 
their needs for the system regularly so that data managers 


Using Data To Improve 
College and Career Readiness 

Measuring the Education Pipeline: 
Common Data Elements indicating 
Readiness. Transition and Success 
provides a brief roadmap to the types 
of questions, data elements and 
conversations that state policymakers 
need to have to inform their efforts to 
improve their state's college and career 
readiness agenda. 


can respond to user demand. Without this constant 
communication and expected calibration, the partners may 
lack credibility and users may not trust the capacity of the 
system. 

As states work to meet the American Recovery and Rein- 
vestment Act (ARRA) requirement to follow individuals 
through the P-20 pipeline and into the workforce by 2011, 
their progress will be expedited and supported by using the 
following processes to guide the development and imple- 
mentation of their cross-agency data sharing efforts: 

► Prioritize, through broad-based stakeholder input, the 
critical policy questions to drive the development and 
use of longitudinal data systems; 

► Ensure data systems are interoperable within and 
across agencies and sectors by adopting common data 
standards, definitions and language; and 

► Protect personally identifiable information through 
governance policies and practices that promote the 
privacy and security of the information while allowing 
appropriate data access and sharing. 

This paper further explores the need to have a common 
P-20/workforce vision and to develop key policy ques- 
tions to drive the development of state data systems. It also 
provides a set of resources (appendices) to help states begin 
this task. 
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■ Beginning To Connect Policy and Data 


States continue to make progress in building the 10 Essential 
Elements of longitudinal data systems; those who are 
strategically building these P-20 /workforce systems to 
answer key policy questions will be able to leverage those 
investments by having greater transparency, richer measures 
of progress and accountability, and information to drive 
continuous improvement. Every state needs to have the 
detailed conversations about basic business or program- 


specific questions around customers, services, needs, 
frequency of updates, access/ security, etc., that will inform 
the directions of expanded longitudinal data systems. Two 
states. New Mexico and South Carolina, provide case studies 
about the importance of developing and prioritizing a set of 
guiding questions before building longitudinal data systems. 
The following are examples of their individual approaches to 
these data /policy opportunities. 


New Mexico 

Staff from the New Mexico Office of Education Accoimt- 
ability, in conjunction with staff from several other states, 
developed the notion of "killer questions" to describe the 
key policy questions that come up over and over again 
across districts and states when they are provided with 
good data. To begin to identify and prioritize these "killer 
questions," New Mexico developed a template (Appendix 
B) that helped staff categorize the policy questions and then 
develop the corresponding data questions they would need 
to first ask and then answer to help solve a policy problem. 
The figure below represents a sample "killer question" and 
a few data questions that will inform the decisionmaking 
process of a policymaker. 



New Mexico was able to identify the questions it needed to 
answer with its data system because it recognized the need 
to connect those responsible for policy and data and found 


a meaningful way to engage them in a discussion. The tem- 
plate serves as a bridge between two seemingly different 
worlds and enables the data system team to understand the 
uses of their product and the priorities of their key stake- 
holders (i.e., customers). 

South Carolina 

The team charged with building and implementing South 
Carolina's longitudinal data system also realized that the 
design should follow the intended uses of the system, but 
like teams in most states, they had a difficult time engaging 
policymakers in discussions about the data system. As a result, 
the team began to identify the key questions on their own, 
and their experience shines a light on the crihcal need for 
policymaker involvement as well as the incredible amount of 
analytical work involved once the questions are determined. 

The project managers in South Carolina (Appendix C) first 
conducted a landscape review of all the questions they 
could find from a variety of sources (e.g.. National Center 
for Education Statistics, various South Carolina stakeholder 
groups. South Carolina legislation) and then prioritized 
the resulting list of almost 400 questions based on whether 
or not the data needed to answer them were available 
quickly and easily. They narrowed their list down to six 
basic questions and used those questions to drive further 
development of the system. They identified categories, topics 
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and subtopics and tagged each question appropriately. They 
also matched each topic with all of the various types of users 
or "roles" fhat would be interested in the answers to those 
questions, and they identified what level of access those 
users would need (e.g., aggregate, student level, etc.). 


Question: What are the course grades for students enrolled in Advanced 
Placement (AP) courses? 


Category: 

Topic: 

Enrollment 

Subtopic: 

AP 

Role: Top state education agency 
staff, student-level data; classroom 
teacher, limited student-level data 

Students 

Topic: 

Test Results 

Subtopic: 

Course Grades 

Role: Principals, student-level data; 
other school staff, aggregate data 
only 


This type of detailed work is exactly what each state must 
go through to build a data system that can answer the 
questions facing stakeholders while providing role-based 
access and ensuring student privacy. Ideally, policymakers 
would have provided more input into the original list of 
questions to ensure that resources are allocated toward 
the highest priorities of the state. The "killer question" 
template referenced above is a great bridge to help both 
policymakers and data staff begin a conversation around 
these questions. 


■ How To Identify and Prioritize the Critical Policy Questions in Your State 


If states are to truly leverage these powerful data systems, 
they must engage the right people in conversations around 
what the system should be designed to do. The ARRA 
funding provides a unique catalyst for these critical discus- 
sions to finally begin in earnesf. Developing a compefifive 
Race to the Top application as well as the plan required in 
the State Fiscal Stabilization Fund will require significant 
participation from stakeholders across the P-20 /workforce 
spectrum. These aligned funding streams are providing 
states with a strategic opportunity to break down tradi- 
tional silos and engage policymakers and data managers 
in a thoughtful dialogue around using data for continuous 
improvement. 

States cannot build an aligned P-20 / workforce data system 
without first defining the goals of the entire system and the 
questions that the system should be designed to answer. 

If a state data system is not able to answer policymaker 
questions, the data system will lose policymaker support, 
and the state will fail to achieve any real sustainability for 
fhe system. 

Once a state has developed an agreed-upon goal and set 
of critical questions around which to build its longitudinal 


data system, concerted efforts must be made to ensure 
this information can be used while also being protected. 

It is vital to ensure that there are governance structures 
and policies to guide the linking, sharing and accessing 
of information while protecting personally identifiable 
information. Common data definitions and architecture 
are critical to making sure disparate data systems in 
various agencies, sectors, districts and states are able to 
communicate with each other and do so efficiently and to 
appropriately link data. When these key steps are taken, 
states will have richer, more detailed, timely and useful 
information to answer their critical questions. 

In addition to notable state efforts to identify and prioritize 
policy questions, the DQC partners have come together to 
identify questions and rank them based on their priorities 
(Appendix D). Our partners also are beginning to work 
together by sector to determine the list of critical questions 
in the early childhood (Appendix E) and postsecondary 
communities. The appendices of this document provide 
states with a place to start as they develop their state's criti- 
cal policy questions. 
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■ Appendix A: Data Quality Campaign's 28 Policy Questions 


Does your system have the necessary elements to address these key policy questions? 


Predicting Success in Later Grade Levels Need Elements 1, 2, 3, 4* 

1. What is the impact of preschool on later academic achievement (e.g., third grade test results)? 

2. Do the effects of our early interventions "fade out" later? 

3. Are students academically prepared for high school? 

4. Which elementary and middle schools in the state are consistently highest-performing in preparing different 
student populations for high school? 

5. Which elementary and middle schools produce the strongest academic growth among initially poorly-prepared 
students, and among initially well-prepared students? 

Academic Growth Need Elements 1, 3, 4 

6. How many students are achieving at least one year's academic growth every year? 

7. How many of the students who started out below grade level are achieving more than a year's growth? 

Achievement Levels in Eai|f Grades as Indicators of Later Success Need Elements 1, 3, 4, 6, 7, 8, 9 

8. What achievement levels in grades 3-7 indicate that a student is "on track" for later success? 

Impact of Grade-Level Retention Need Elemen ts 1 , 2 , 3, 4, 6, 7, 8, 9 

9. What effect does early grade retention have on later academic success of students who were retained in the early 
grades? 

Course Rigor Need Elements 1 , 3, 6, 7 

10. What 8th grade achievement levels indicate that a student is well prepared to succeed in challenging courses in 
high school? 

11. Have students taken the courseworkto prepare them for college and work — both in years of study and rigor of 
content? 

12. What evidence exists that students who take and pass the courses have learned the course content? 

Sustaining Enrollment in Earif Grades Need Elements 1, 2 

13. What students are being lost in transition between middle and high school? 

14. What proportion of the students who enter elementary school maintain continuous enrollment and complete 8th 
grade in a timely manner? 

Consistentlf High-Performing Schools Need Elements 1, 6 

15. Which elementary and middle schools in the state are consistently highest-performing in preparing different 
student populations for high school? 

College Preparation Need Elements 1 , 3, 6, 7, 8, 9 


16. Are the students academically prepared to graduate from high school and enter college? 
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High School Indicators of College Preparedness Need Elements 1, 3, 7, 9 

17. What high school achievement levels indicate that a student is college and work ready? 

18. Are students academically prepared to enter college and complete their program or degree in a timely manner? 

19. What is the relationship between students' performance on state assessments (high school exit exam, end-of- 
course exams) and subsequent postsecondary performance and graduation? 

College Remediation Need Elements 1, 8, 9 

20. What percentage of high school graduates who go on to college take remedial courses? 

High School Completion Rates Need Elements 1, 2, 8 

21. What proportions of the students who enter 9th grade maintain continuous enrollment and complete their high 
school requirements in a timely manner? 

High Performing Schools: Collage Pregaration of SubgrouDfi Need Elements 1, 2, 3, 7, 9 

22. Which high schools in the state are consistently highest-performing in preparing different student populations for 
college and work? 

Academic Growth bf Prior Performance Subgroup Need Elements 1, 2, 3, 7 

23. Which high schools produce the strongest academic success for initially poorly-prepared students, and for initially 
well-prepared students? 

College Success of K-12 Students Need Elements 1, 9 

24. In what content areas do students require remediation? 

25. What are the retention and degree completion rates of students who are placed in remedial coursework? 

Dual Enrollment Need Elements 1, 6, 7, 9 

26. How do dual-enrollment and advanced placement programs in high school affect students' success in college? 

Graduation Rates bf Subgroup and Prior Performance Need Elements 1, 2, 3, 8 

27. Which institutions are doing the best job of graduating students on time, based on those students' prior 
preparation and level of economic disadvantage? 

Teacher Effectiveness and Preparation Programs Need Elements 1, 3, 4, 5 

28. Which teacher preparation programs produce the graduates whose students have the strongest academic 
growth? 

NGA Graduation Rate Need Elements 1, 2,8,10 


*The 1 0 Essential Elements are: 

1. A unique statewide student identifier that 
connects student data across key databases 
across years 

2. Student-level enrollment, demographic and 
program participation information 

3. The ability to match individual students' 
test records from year to year to measure 
academic growth 


4. Information on untested students and the 
reasons they were not tested 

5. A teacher identifier system with the ability to 
match teachers to students 

6. Student-level transcript information, including 
information on courses completed and grades 
earned 


7. Student-level college readiness test scores 

8. Student-level graduation and dropout data 

9. The ability to match student records between 
the P-12 and postsecondary systems 

10. A state data audit system assessing data 
quality, validity and reliability 
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■ Appendix B: "Killer Questions" Template 

Source: New Mexico Office of Education Accountabiiity 

This template can be used by stakeholders representing both policy and data to bridge their traditionally siloed work and 
begin a conversation about data use in their states. The questions listed below are examples. A more comprehensive template 
is found at http:/ /dataqualitycampaign.org /resources /447 . 


Question Type 

Data Questions 

Policy Questions 

Political Questions 

Policy Elements 

Data for Advocacy and 
Accountability 

Statute, Regulation, Standards, 
Frameworks, Budget 

Power Groups, Practices 
& Behaviors 

Example 1 

• What percentage of students 
score proficient or above on 
achievement tests in math or 
reading? 

• What are the achievement 
gaps among different groups of 
students? 

• How should the certification 
requirements for teachers be 
strengthened? 

• What kinds of instructional interventions 
should be implemented? 

• How will these reforms and interventions 
be funded and where will the money 
come from? 

• Who has the influence to change how 
teachers are prepared? 

• What groups control professional 
development? 

• Who controls where the most effective 
teachers and principals are placed? 

• Who evaluates the teachers? 

• Who can change how resources are 
allocated? 

Example 2 

• What percentage of ninth 
graders enter college four years 
later? 

• What percentage of high school 
graduates take remedial courses 
in college? 

• What changes do we make in high 
school standards and teacher training 
to ensure that more students graduate 
ready to succeed in college? 

• What statutory changes should be 
enacted to better align the curriculum 
between high school and college? 

• How will these initiatives be funded and 
where will the money come from? 

• Who has the influence to change prac- 
tices and behaviors in high school? 

• Who has the power to get colleges and 
high schools to talk to one another? 

• Who has the influence to get high 
schools and colleges to cooperate 
rather than compete for limited 
resources? 
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South Carolina took each of its prioritized questions and began a process that included rewording the questions and tagging 
each with categories, topics, subtopics and keywords. Following are draft materials supplied by the South Carolina Depart- 
ment of Education. The materials are not final. 






Original QUESTIONS 


Category(ies) 

Topic(s) 

Sub-Topic(s) 

Keywords 

for Data Warehouse 

New and Reworded Questions 

Students 

Enrollment 

AP 

AP 

1 5. What is the Advanced 

1 5. What is the enrollment in AP classes per 




Enrollment 

Placement (AP) 

grade per school per district per state? 




enrollment? 

What are the subgroups enrollments 




Grade-level 


in AP classes per grade per school per 




School-level 


district per state? 




District-level 






State-level 



Students 

Enrollment, 

AP 

AP 

1 6. What is the Advanced 

1 6. What are the course grades for students 


Test Results 

Course Grades 

Students 

Placement (AP) 
enrollment by test 

enrolled in AP courses? 




Course-Grades 

scores? 


Educators, 

Performance 

Student 

Teacher 

51. What is the teacher to 

51. Provide teacher rosters showing only 

Students 


Performance 

Student 

student connection 

students registered for an individual 




(available to districts, if 

teacher's courses/classes. Provide course- 




Student-Roster 

not LEAs)? 

grades, absences, tardies, disciplinary 




Course-Grades 


actions, as of last data collection. 




Student-Attendance 






Student-Absence 






Student-Tardies 






Discipline 



Administration 

School, 

Capacity 

Student-Enrollment 

55. Enrollment reports: 

55. Provide school-, district-, and state-level 


District, 

Planning 

Actual-Enrollment 

projections, numbers. 

enrollment reports with current and 


State 


student demographics 

future projections. Include student 
demographics. Base projections on 
history and available trend information. 
Provide subtotals within each level 

Enrollment-Projections 

Projected-Enrollment 




Enrollment-Trends 


of projection (demographics within 




School-Enrollment 


grade-level within school; demographics 
within grade-level & school & district; 




District-Enrollment 


demographics within grade-level 




State-Enrollment 


& school & district & state). Provide 
flexibility to see desired level only. 
Provide drill-down capabilities. 


Share: c EaiD>s. 


Data Quality Campaign | CONNECTING POLICY AND DATA 


I 




Category 

Topics 

Sub-Topics 


Leadership 

Staff Mobility 

School Profile 

District Profile 

State Profile 

Student 

Mobility 

Aggregate 

Student 

Performance 

Drop Out Rate 




District 

Curriculum 

Goals 

Leadership 

Issues 

Climate 

Priority Schools 

Drop Out Rate 

Capacity 

Planning 



Administration 

School 

Curriculum 

Goals 

Leadership 

Issues 

Climate 

Priority Schools 

Drop Out Rate 

Capacity 

Planning 




State 

Curriculum 

Goals 

Leadership 

Issues 

Climate 

Priority Schools 

Drop Out Rate 

Capacity 

Planning 




Program 

Effectiveness 

Goals 

Programs 

AP 

IB 

Reading first 






Staff Mobility 

Programs 

Principals 

Teachers 

frequency 

Attrition 

Retention 





Certifications 

HQ 

Professional 

Development 

Credentials 

ADEPT 

Technology 

Proficiency 





Educators 

Mobility 

Recruitment 

Retention 

Tenure 

Attrition 







Performance 

Student 

Performance 

Teacher 

Attendance 

IGP 

Completion 

Rate 

Attendance 

Vacancies 

Indicators 





Achievement 

Test Results 
(PACTHSAP) 

Goals 

Assistance 

Gifted & Talented 

Programs 

Indicators 

Student Groups 

Patterns 



Adult Ed 

GED 










Attendance 

Truancy 

Tardiness 

Absences 

Home Bound 







Career 

Assessments 

Individual 
Graduation Plan 

College Choice 









Demographics 

Socioeconomic 

Gender 

Ethnicity 

Age 






Students 

Discipline 

Absentee 

Patterns 

Suspensions 

Expulsions 

Interventions 

Referrals 




Enrollment 

Registered 

Students 

No Shows 

Exiting 

Students 

AP 

IB 

Projections 





Graduation 

Areas of Study 

Post-Grad 

Plans 

Rates/ 

Reasons 








Mobility 

Exiting Students 

Entering 

Students 

Classroom 

School 

District 






Participation 

Programs 

(AP.IB) 

Courses 

Completed 

Course Grades 

Extracurricular 

Activities 

School 

Choice 

Extended 

Learning 

Certifications 

Indicators 

Charter 

Schools 


Performance 

Retention 

Drop-Outs 

At-Risk 

Models 

Course Grades 







Test Results 

PACT /PASS 

HSAP 

EOCEP 

Local Diagnostics 

SC-Alt 

ELDA 

Other Tests 

Course 

Grades 

Student 

Groups 
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South Carolina further tagged each topic by types of users and identified the type and level of data each would be able to 
access to ensure student privacy. 


STATE 

DISTRICT 

SCHOOL 

TOPICS* 

SEA Top 
Level (S1) 

Other SEA 
Staff (S2) 

LEA Top 
Level 
(LI) 

Other LEA 
Staff (L2) 

Principal 

(PI) 

Guidance 

(G1) 

Classroom 

Teacher 

(Cl) 

Subject/ 

Program 

Coordinators (U1) 

Other 

School Staff 
(U2) 

Adult Ed 

✓ 

A 

V 

A 

■/ 

✓ 

■/ 

■/ 

A 

District Performance 

✓ 

A 

V 

A 


✓ 

✓ 

■/ 

A 

Graduation 

✓ 

A 

✓ 

A 

✓ 

✓ 

•/ 

V 

A 

Post-Secondary 

✓ 

A 

✓ 

A 

✓ 

✓ 

•/ 


A 

Program Effectiveness 

V 

A 

V 

A 

■/ 

✓ 

✓ 

■/ 

A 

Schooi Performance 

V 

A 

V 

A 


✓ 

•/ 

■/ 

A 

Staff Mobiiity 

✓ 

A 

•/ 

A 

✓ 

✓ 

•/ 


A 

Student Achievement 

✓ 

A 

•/ 

A 

•/ 

✓ 

•/ ♦ 

V' 

A 

Student Attendance 

V 

A 


A 

■/ 

✓ 

♦ 

■/ 

A 

Student Career Assessments 

✓ 

A 

•/ 

A 

✓ 

✓ 

♦ 

✓ 

A 

Student Demographics 

V 

A 

•/ 

A 

✓ 

✓ 

♦ 

■/ 

A 

Student Discipline 

V 

A 

•/ 

A 

✓ 

✓ 

♦ 

■/ 

A 

Student Enrollment 

✓ 

A 

•/ 

A 

✓ 

•/ 

♦ 

■/ 

A 

Student Mobility 

✓ 

A 

•/ 

A 

✓ 

✓ 

♦ 

■/ 

A 

Student Participation 

V 

A 

V' 

A 


■/ 

♦ 

■/ 

A 

Student Performance 

V 

A 

V' 

A 

V' 

■/ 

♦ 

■/ 

A 

Student Test Results 

✓ 

A 

•/ 

A 

✓ 

✓ 

♦ 

✓ 

A 

Teacher Certifications 

V 

A 

V 

A 

✓ 

V' 

•/ 

■/ 

A 

Teacher Performance 

■/ 

A 

■/ 

A 

✓ 

A 

A 

■/ 

A 

* See Glossary for 
definitions ofTopics 

All data; 
Includes IT 

Same as 
Public; 
Aggregate 
Only 

Student- 
level by 
class & 
above 

Same as 
Public; 
Aggregate 
Only 

Student- 
level by 
class & 
above 

Student- 
level by 
class & 
above 

Only 
Students 
in classes 


Same as 
Public; 
Aggregate 
Only 


•/ 

Student-level Data 

A 

Aggregated Data Only 

A 

Limited Access 

♦ 

Limited to Students In Teacher's Classes 

I &A 

Individual & Aggregate Data 
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■ Appendix D: Survey of DQC Partners on Critical Questions for States 


There is a demonstrated need for some consensus across 
stakeholders as to the critical policy questions that would best 
drive the development of state data systems linking P-20 
and workforce data and simultaneously inform efforts to 
develop tools and resources that support these state-level 
efforts (e.g., common data standards). 

States such as New Mexico and South Carolina have put 
tremendous effort into researching and identifying criti- 
cal questions. Following their example and as a response 
to increasing state requests on this issue, the DQC pulled 
together questions from these and other states and compiled 
a list of 57 that we hear discussed the most (see pages 12-15). 
We then asked our state and national partners to rate them 
as high, medium or low priority. 

HIGH: These questions are critical to improving student 
achievement, and statewide data systems must be able 
to answer them as soon as possible. 

MEDIUM: These questions are important to improving 
student achievement, and statewide data systems need 
to be able to answer them in the near future. 

LOW: These questions are related to improving student 
achievement, and statewide data systems should be able 
to answer them at some point. 


The results represent the views of only the 24 partners/ 
states that completed the survey and should be viewed 
within that narrow context. The DQC primarily works with 
organizations that prioritize issues that are the responsi- 
bility of state education agencies and center most around 
K-12; therefore, the results reflect a higher priority on 
these issues, and the respondents placed a lower priority 
on workforce, early childhood and postsecondary-related 
questions. 

This survey primarily serves as an example of how to iden- 
tify and prioritize the questions that are most important to 
your state. Because our partners (e.g.. National Governors 
Association, Council of Chief State School Officers, State 
Higher Education Executive Officers, etc.) are representa- 
tive of the stakeholders that should be at the table in your 
state, use these questions as a beginning to bring together 
your stakeholders and have them similarly rate the ques- 
tions and add new questions that may not be part of the 
initial list. Once you have a set list of questions, we encour- 
age you to further the conversation using the resources 
identified elsewhere in this paper (e.g., "Killer Questions" 
Template, Measuring the Education Pipeline, etc.) to help your 
policymakers and data managers determine how to best 
build a data system that answers the identified questions in 
a prioritized manner. 
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DQC Critical Policy Questions Survey Results 

K-12 

Teachers 

1 . What is the distribution of effective teachers and 
principals in the state? 

High Medium Low 


61% 26% 13% 


2. What percent of students are taught by HQT teachers? 
Effective teachers? 

High Medium Low 


48 % 39% 13% 


3. How are HQT teachers distributed among and within 
schools? Effective teachers? 

High Medium Low 


64 % 27% 9% 


4. How does the percentage HQT teachers relate to student 
success? Effective teachers? 

High Medium Low 


87 % 9% 4% 


5. How can teacher preparation and professional 
development be improved so that all teachers are 
effective? 

High Medium Low 


65% 22% 13% 


6. Which institutions consistently produce more effective 
teachers and what are the program components that are 
most linked to effective teachers? 

High Medium Low 


83 % 13% 4% 


7. What is the value-add of advanced teacher education 
(i.e., advanced degrees) in terms of student achievement? 

High Medium Low 


30 % 43% 26% 


8. What is the value-add of educator professional 
development in terms of student achievement? 

High Medium Low 


52 % 35% 13% 


Achievement 

1. What percentage of students in all subgroups score 
proficient or above on achievement tests in math and 
reading? 

High Medium Low 


70 % 17% 13% 


2. Are students achieving at least one year's academic 
growth every year? 

High Medium Low 


74 % 22% 4% 


3. What 8th grade achievement levels indicate that a 
student is well prepared to succeed in challenging 
courses in high school? 

High Medium Low 


70 % 26% 4% 


4. What evidence exists that students who take and pass 
the courses have learned the course content? 

High Medium Low 


52 % 26% 22% 


5. What is the percentage of high school graduates that 
complete Algebra II or equivalent? 

High Medium Low 


43 % 52% 4% 


6. Which elementary and middle schools in the state are 
consistently highest-performing in preparing different 
student populations for high school? 

High Medium Low 


39 % 48% 13% 


7. Which elementary and middle schools produce the 
strongest academic growth among initially poorly- 
prepared students, and among initially well-prepared 
students? 

High Medium Low 


78 % 13% 9% 
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8. What achievement levels in grades 3-7 indicate that a 
student is "on track" for later success? 

High Medium Low 


70 % 26% 4% 


9. Which high schools produce the strongest academic 
success for initially poorly-prepared students and for 
initially well-prepared students? 

High Medium Low 


83 % 13% 4% 


10. How are students performing on college prep tests 
including ACT, SAT, AP? 

High Medium Low 


61 % 30% 9% 


At Risk 

1. What are the indicators that serve as early warning that 
students are at risk of dropping out? 

High Medium Low 


83 % 13% 4% 


2. What percentage of schools serving a significant at-risk 
population has demonstrated success as determined by 
graduation rate? 

High Medium Low 


52 % 35% 13% 


3. What students are being lost in transition between 
middle and high school? 

High Medium Low 


74 % 22% 4% 


4. What dropout intervention strategies are in use and how 
are they being assessed for effectiveness? 

High Medium Low 


57 % 39% 4% 


Early Childhood Linkage 

1. To what degree does participation in early childhood 
programs (e.g., Head Start, Early Head Start, state Pre-K, 
etc.) increase kindergarten readiness? Are these gains 
sustained through third grade? Do results vary by race/ 
ethnic and/or socio-economic groups? 

High Medium Low 


61 % 30% 9% 


5. What is the graduation and dropout rate by subgroup? 

High Medium Low 


70 % 22% 9% 


6. What effect does early grade retention have on later 
academic success of students who were retained in the 
early grades? 

High Medium Low 


30 % 48% 22% 


Other 

1. What percentage of students is properly immunized? 

High Medium Low 


22 % 26% 52% 


2. What is the absentee rate in each school? 

High Medium Low 


39 % 48% 13% 


3. What percentage of students received vision/hearing 
screenings in school? 

High Medium Low 


13 % 43% 43% 


4. What can we determine about student achievement by 
analyzing discipline data? 

High Medium Low 


17 % 70% 13% 


5. What is the amount and source of funds going to 
economically disadvantaged students? ELL students? 
Schools not meeting AYP? 

High Medium Low 


46 % 38% 17% 


2. What is the impact of universal state pre-K on students' 
kindergarten readiness? Are these gains sustained 
through the third grade? 

High Medium Low 


43 % 43% 13% 
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3. Does participation in early childhood programs improve 
student future achievement (e.g., scoring proficient 
or above on state exams, graduation, enrollment in 
postsecondary education, etc.)? 

High Medium Low 


58% 29% 13% 


4. Which early childhood programs are doing the best 
preparing students for kindergarten, based on those 
students' prior preparation and level of economic 
disadvantage? 

High Medium Low 


54% 33% 13% 


5. How does the length of enrollment in early childhood 
program(s) affect kindergarten readiness and future 
achievement? 

High Medium Low 


30% 57% 13% 


Postsecondary Linkage (questions in bold also include workforce linkage) 


1. How many of our state's high school graduates need 
remediation in college? What is the college-completion 
rate for these students? 

High Medium Low 


83% 17% 0% 


2. What is the college going rate for our high school 
graduates? College completion? 

High Medium Low 


83% 8% 8% 


3. To what degree do our state financial aid programs 
improve college access? 

High Medium Low 


43% 39% 17% 


4. What high school performance indicators (e.g., 
enrollment in rigorous courses or performance 
on state tests) are the best predictors of students' 
success in college or the workplace? 

High Medium Low 


83% 9% 9% 


5. Do students who earn college credit in high school more 
likely to go to college? Are they more likely to graduate 
from college on-time? 

High Medium Low 


48% 43% 9% 


6. Have students taken the coursework to prepare 
them for college and work — both in years of study 
and rigor of content? 

High Medium Low 


65% 35% 0% 


7. What is the relationship between students' performance 
on state assessments (high school exit exam, end- 
of-course exams) and subseguent postsecondary 
performance and graduation? 

High Medium Low 


65% 30% 4% 


8. Which high schools in the state are consistently 
highest-performing in preparing different student 
populations for college and work? 

High Medium Low 


61% 26% 13% 


9. How do dual-enrollment and advanced placement 
programs in high school affect students' success in 
college and the workplace? 

High Medium Low 


59% 36% 5% 


10. Which institutions are doing the best job of graduating 
students on time, based on those students' prior 
preparation and level of economic disadvantage? 

High Medium Low 


61% 26% 13% 
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11. What changes in curriculum and teacher training will 
reduce the need for remediation? 

High Medium Low 


57% 39% 4% 


12. Do patterns in course taking result in success on state 
graduation tests and college entrance exams? 

High Medium Low 


57% 35% 9% 


13. Where do our students go after high school? 

High Medium Low 


74% 22% 4% 



14. Does our state scholarship program increase student 
participation in postsecondary education within the 
state? How many attend out-of-state institutions with 
state scholarships? 

High Medium Low 


50% 38% 13% 


15. What percentage of 9th graders enter college four years 
later? 


High Medium Low 


70% 26% 4% 


Workforce Linkage 

1. What percentage of our high school graduates (who 
haven't obtained a college credential) continue to live 
and work in our state? 

High Medium Low 


61% 26% 13% 


2. What percentage of our high school graduates, including 
those who earned GEDs, has some postsecondary 
education? On average, how many more credits do these 
individuals need to obtain postsecondary credentialing? 

High Medium Low 


65% 22% 13% 


3. What percentage of our workforce population return 
to postsecondary education for additional training, but 
without earning a credential? 


4. What percentage of our college graduates continue to 
live and work in our state? 

High Medium Low 


65% 17% 17% 


5. How much do our high school and college graduates 
earn in the workforce over time? What about our drop- 
outs? 

High Medium Low 


65% 30% 4% 


6. How many of our high school and college students are 
employed while they are in school and what kind of 
impact does it have on their academic success? 

High Medium Low 


35% 39% 26% 


7. Which industries employ the majority of our state's high 
school and college graduates? 

High Medium Low 


64% 27% 9% 


How do we strengthen the relationship between 
education and the economy? 

High Medium Low 


57% 30% 13% 


High 

Medium 

Low 

52% 

39% 

9% 
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■ Appendix E: Policy and Practitioner Questions Using Early Childhood Data 

The following list is based on a series of "killer questions" generated at a July 2009 Early Childhood Data Working Meeting 
that was convened by the Early Childhood Data Collaborative (ECDC),* subsequent presentations of the questions and collec- 
tive knowledge of the ECDC. The questions are loosely organized by stakeholder group (policymakers and practitioners) and 
offer examples of data that could be collected and used to answer the questions. This group is currently working to refine and 
validate the questions and related data elements. 


Question that 

resonates with Question that resonates with 

state policymakers early childhood practitioners Examples of data needed to answer these questions 


Are children, birth 
through age 5, on 
track to succeed at 
school entry? 

How well are young children progressing 
in all aspects of development from birth 
to 3rd grade? 

Who are the children making the most 
progress? 

Data on all children disaggregated by: 

• Risk factors — race, income, dual language learners, single parents, unemployed 
parents, etc. 

• Age — 0-2, 3-5, kindergarten entry, 3rd grade 

Data on child demographics linked to measures of developmental progress: 

• Social/emotional, physical, cognitive, linguistic, approach to learning (or 
knowledge, skills and behavior) 

Which children 

Where are the children receiving services? 

Data elements from above linked to: 

have access to 


• Dosage data — how many hours per day, days per week, years is the child 

high-quality early 

Are we meeting the needs of at-risk 

receiving (not the program offered) 

care and education 

children? 

• Program location — number of slots/supply by setting 

programs? 


• Provider characteristics — age, ethnicity, language, educational level, certification/ 


Where are the programs that offer 

credential, experience in the field 


services? 

• Quality standards (research-based, statewide) that include — program standards, 
teacher standards, child outcomes (early learning standards or guidelines) or 


Who are the providers? 

What is the quality of the programs? 

developmental benchmarks 

Is the quality of 

What program characteristics are 

Data elements from above linked to program characteristics and tracked over time. 

the programs 

associated with positive outcomes? 

Examples of program characteristics include: 

improving? 


• Structural measures — teacher or care provider/child ratio, class size, teacher 


How well are various types of programs 

credential 


and local agencies improving outcomes 

• Quality measures — Quality Rating & Improvement System rating, environmental 


for different types of children? 

safety rating or score, consistent use of curriculum, Classroom Assessment Scoring 
System score 

What is the cost 

What public and private funds are spent 

Data on program quality linked to cost information, such as: 

per child of a high- 

on early care and education programs? 

• State expenditures 

quality program? 


• Long- and short-term cost savings: special education costs, retention. 


What proportion of state funding is being 
spent on high-quality programs? 

remediation, dropout prevention 

What policies and 

What practices and policies best build 

Data elements for workforce: 

investments lead 

and sustain the early childhood workforce 

• Compensation 

to a skilled and 

necessary for quality early childhood 

• Benefits 

stable early care 

education programs that promote positive 

• Retention 

and education 

child outcomes? 

• Professional preparation and ongoing professional development (degree. 

workforce? 


what degree in, practicum, mentoring) 


Do all providers and teachers (or all 

• Data linked back to workplace and working conditions at the workplace 

How prepared is 

members of the workforce) have access 

(including turnover rates) 

the workforce to 

to appropriate education and ongoing 


provide effective 

professional development? 

Data elements for the professional development system: 

education and care 


• Noncredit, associate, bachelor's, master's programs available 

for all children? 


• Content and structure of the program — curriculum, practicum, student supports 

• Faculty characteristics 


*A partnership of the Council of Chief State School Officers • Center for the Study of Child Care Employment at UC Berkeley • Data Quality Campaign ■ National 
Center for Children in Poverty • National Conference of State Legislatures • National Governors Association Center for Best Practices • Pre-K Now at the Pew Center 
on the States 
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DATA UALITY 

CAMPAIGN 


Using Data To improve student Achievement 


www.DataQualityCampaign.org 


The Data Quality Campaign (DQC) is a national, collaborative effort to 
encourage and support state policymakers to improve the availability and 
use of high-quality education data to improve student achievement. The 
campaign will provide tools and resources that will help states implement and 
use longitudinal data systems, while providing a national forum for reducing 
duplication of effort and promoting greater coordination and consensus among 
the organizations focused on improving data quality, access and use. 


Managing partners of the Data Quality Campaign include; 


► Achieve. Inc. 

► Alliance for Excellent Education 

► Council of Chief State School Officers 

► Education Commission of the States 

► The Education Trust 

► National Association of State Boards 
of Education 


► National Association of System Heads 

► National Center for Educational 
Achievement 

► National Center for Higher Education 
Management Systems 

► National Conference of State Legislatures 

► National Governors Association Center 
for Best Practices 


► Schools Interoperability Eramework 
Association 

► State Educational Technology Directors 
Association 

► State Higher Education Executive 
Officers 


Endorsing partners of the Data Quality Campaign include; 


► 3 Rivers Connect 

► Academy for Educational Development 

► ACT 

► Alliance for Quality Teaching 

► American Association of Colleges for 
Teacher Education 

► American Association of Community 
Colleges 

► American Association of State Colleges 
and Universities 

► American Board for Certification of 
Teacher Excellence 

► American Federation of Teachers 

► American Productivity and Quality 
Center 

► American Society for Quality 

► American Youth Policy Forum 

► Arise Citizens' Policy Project 

► Association for Career and Technical 
Education 

► Business Higher Education Forum 

► Campaign for High School Equity 

► Center for Public Policy Priorities 

► Center for Teaching Quality 

► College Summit 

► Complete College America 

► Consortium for School Networking 

► Educational Policy Institute 

► ElS 

► Forum for Youth Investment 

► Foundation for Excellence in Education 


► Great Schools 

► Institute for a Competitive Workforce 
(An Affiliate of the U.S. Chamber of 
Commerce) 

► Institute for Educational Leadership 

► Institute for Higher Education Policy 

► International Association for K-12 Qnline 
Learning 

► Tames B. Hunt. Tr. Institute for 
Educational Leadership and Policy 

► lobs for the Future 

► Knowledge Alliance 

► League of Education Voters Foundation 

► Learning Point Associates 

► Maine Center for Economic Policy 

► Midwestern Higher Education Compact 

► Military Child Education Coalition 

► National Alliance for Public Charter 
Schools 

► National Association for the Education of 
Young Children 

► National Association of Early Childhood 
Specialists in State Departments of 
Education 

► National Association of Secondary 
School Principals 

► National Association of State Directors of 
Teacher Education and Certification 

► National Association of State Workforce 
Agencies 


► The National Center for Public Policy and 
Higher Education 

► National Council for Accreditation of 
Teacher Education 

► National Council on Teacher Quality 

► National Math and Science Initiative 

► National School Boards Association 

► National Staff Development Council 

► National Student Clearinghouse 

► Nebraska Appleseed Center for Law in 
the Public Interest 

► New England Board of Higher Education 

► Northwest Evaluation Association 

► Pathways PA 

► Pathways to College Network 

► Pell Institute 

► Policy Innovators in Education Network 

► Postsecondary Electronic Standards 
Council 

► Pre-K Now 

► Public Education Network 

► Roads to Success 

► Southern Regional Education Board 

► Statewide Poverty Action Network 

► The Workforce Alliance 

► Thomas B. Fordham Institute 

► Western Interstate Commission for 
Higher Education 

► The Working Poor Families Project 
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